Estimating the number of components with defects post-release that showed no defects in testing

Components that have defects after release, but not during testing, are very undesirable as they point to ‘holes’ in the testing process. Either new components were not tested enough, or old ones were broken during enhancements and defects slipped through testing undetected. The latter is particularly pernicious, since customers are less forgiving when existing functionality is no longer working than when a new feature is not working quite properly. Rather than using capture–recapture models and curve-fitting methods to estimate the number of remaining defects after inspection, these methods are adapted to estimate the number of components with post-release defects that have no defects in testing. A simple experience-based method is used as a basis for comparison. The estimates can then be used to make decisions on whether or not to stop testing and release software. While most investigations so far have been experimental or have used virtual inspections to do a statistical validation, the investigation presented in this paper is a case study. This case study evaluates how well the capture–recapture, curve-fitting and experience-based methods work in practice. The results show that the methods work quite well. A further benefit of these techniques is that they can be applied to new systems for which no historical data are available and to releases that are very different from each other. Copyright  2002 John Wiley & Sons, Ltd.